Evaluating performance in systems with heavy tailed input: a quantile based approach

One of the key invariants in computer and communication systems is that im¬ portant characteristics follow long-or heavy-tailed distributions. This means that the tail of these distributions declines according to a power law. Hence, the probability for extremely large values is non-negligible. For example, such distributions have been found to describe the size of web objects or the pro¬ cessing latencies in computer and communication systems. As a consequence, there is a need to employ such distributions when evaluating such systems with synthetic workloads. However, sampling from such distributions to generate workloads implies that the system under evaluation remains in transient state over all periods of time that are feasible for performance evaluations. Con¬ sequently, frequently-used statistics for performance evaluation, such as the average of the system output, do not converge. In this thesis we move away from evaluation using statistics such as the average, which describe the expected behavior of the system in all cases, and take the step towards evaluation using statistics such as quantiles, which de¬ scribe the behavior in a given percentage of cases. Such quantiles of the sys¬ tem output do not depend on the extreme tail of the output distribution. We therefore address the problem of whether employing quantiles can enable per¬ formance evaluations within periods of time that are feasible in practice for performance evaluations. Quantiles have a natural interpretation to statistically characterize the sys¬ tem performance. If e.g. a system offers a web service, the 99-th percentile of the latencies of downloads can statistically characterize the system perfor¬ mance from a user's view, since 99% of downloads terminate within times smaller than this quantile. If converged, such quantiles can be used to derive statistical guarantees for the system performance. Similar statements hold for l 11 Abstract system components such as servers and networks. Applying probability theory, we show that statistics of quantiles converge considerably faster than other frequently-used performance evaluation statis¬ tics if the underlying distribution is long-or heavy-tailed. Based on this theory, we give a method which enables to evaluate system performance under long-or heavy-tailed input within periods of time that are practically feasible. We validate the proposed method by applying it to a simulation-based evaluation of the network performance of systems that offer web services. We show that the proposed method has further applications to other prob¬ lems that require performance evaluation with synthetic workloads which are generated by sampling …